9.1 K-means Algorithm for Clustering

and showed the solution G is the leading eigenvectors of the symmetric positive semi definite matrix K. When K = AA> (sample covariance matrix) with A = [x1, ...,xm], xi ∈ Rn, those eigenvectors form a basis to a k-dimensional subspace of Rn which is the closest (in L2 norm sense) to the sample points xi. The axes (called principal axes) g1, ...,gk preserve the variance of the original data in the sense that the projection of the data points on the g1 has maximum variance, projection on g2 has the maximum variance over all vectors orthogonal to g1, etc. The spectral decomposition of the sample covariance matrix is a way to ”compress” the data by means of linear super-position of the original coordinates y = G>x. We also ended with a ratio formulation: